LTX2: Efficient Joint Audio-Visual Foundation Model

LTX2 is a foundation model for text-to-audio-video (T2AV) and text-image-to-audio-video (TI2AV) with the input image being the first frame of video.

Table of Contents

1. Video and Audio Latent Space

LTX2 follows the latent diffusion architecture. It decouples the latent spaces of videos and audios. Speaking of latent spaces, it requires an encoder to map videos or audios into a latent tensor and a decoder to map the latent tensor back into videos or audios.

1.1. Video VAE

1.1.1. Spacial and Temporal Compression

Videos need to be compressed at 3 dimensions: height, width, and time1. Therefore, a video VAE encoder maps a video of shape \([B,3,T,H,W]\) into a latent tensor of shape \([B,C,t,h,w]\), where \(B\) is the batch size; \(T,H,W\) are duration (the number of frames), frame height, frame width respectively; \(C\) is the channel size in latent space (usually larger than \(3\) to capture more features); \(t,h,w\) are downsampled duration, height, and width by a certain factor such that \(t=T/r_{t}, h=H/r_{s}, w=W/r_{s}\)2

1.1.2. Casual VAE

During convolution, kernels are only allows to inspect current and prior frames, to avoid information leakage from future frames.

1.1.3. Positional Embedding for Video Latent Space

As a video has 3 dimensions, therefore, 3D RoPE is applied to \(t,h,w\) before the latent tensor is sequentialized and sent to Transformer blocks.

1.2. Audio VAE

Footnotes:

1

The intuition is that, nearby frames may contain redundant information, therefore they can be compressed.

2

Note here, factors are the same for height and width, but not for the time-axis.

Date: 2026-09-27 Sun

Author: ArcaLunar